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ABSTRACT 

Members of the beta family of distributions were used 
to estimate percentile ranks and to accuaulate normative data 
collected in a university-wide system for gathering student opinions 
about teaching — including the areas of course contentr objectives r 
instructor's behavior^ teaching methods and materialSr and outcom*5s 
cf instruction. The fitted distributions^ were found to te acceptably 
accurate and to provide as accurate ci mere accurate percentile ranks 
than the linear interpolation method often used for this purpose. In 
additioRr the estimation method investigated requires storage of only 
minimal data in order to accumulate (update) norms from year to year. 
.The nature of the results suggests that the beta family may be a 
useful modeling distribution for norirs ci population distribution 
estimates on Likert-type items in oth'^r attitude and opinion research 
areas. (Author/MH) 
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-3ae .1^. . -opiaent c f :^nns foi -siy asssHBEni asm evaluation instru- 
ments b-?OTO=^r=i^ ^vo; '=;^ ^he use relatively fev observations and tire 
resu-c^ r«p±rii:al nores (e.g., ^>,oercentiles ) of «st show irregular "jumps" 
at ta«t 3»r ,c»tr ends ux ^ scor^ sale. Th±s issports on the use 

matirtinfc ■-ed£-E to ^^ove a- erapiri :al nor* im a situation where 

the asp ■ ' dls=3bur=ms car ^ expec -ed to ^ tettgfaly skewed and for 
which '- servations care ini ..tl±-Lly available in the "light" tail. 

-f the beta family - distributions wmk used to estimate 
P^ T- — IV rajiks and tc accumulate normative data collected in a univer- 
sity-- ^ys^em for gathering student opinions aiiout teaching. The fit^ 
ted dlacrilnutlons were found t» be acceptably accurate and to provide as 
accnratEJor ^ n-e accurate per>iH2itile ranks than the linear interpolation 
method f-±as used for this purrpDse. In addition, the estimation method 
investigansttl ^requires stocagfc to; only minimal data in order to accumulate 
(update) n^mas from year to s^a:^. The nature of the results suggests that 
the beta-^^a: -y may be a usfsi'ul inodeling distribution for norms or popu- 
lation &teEni.bution estimatfis^ an Likert-type items in other attitude and 
opinion - -Jiwirch areas. 



nSING BETA DTSTlftlBUTIONS'TO ESTIMATE 
tmCSSSTLl BANKS Am i^ClMTLAlEE: NOBMS FOR STUDENT 
OPINIOB^ TEACHING ITEMS 



Questilonxiai; iss for collecrfmg student opinions about teaching are used 
QV" jiost college CBflq>uS4e8. Most systems £or^ processing such ratings tabu- 
.^te Indlvidtual student ratings cm each Ite^ and report class means (med- 
.ians) to the I ngrr uctor. Many ^stems alsc report percentile ranks so that 
moan raiclm^ by a -class or a :?iHXtlcular :l.teB^nsay be normatlvely compared 
:t tite DXtftth raiting of an axoEtspriate population of classes. 

31wo TsuB^^yuyds f^x arriving sec the percentllj^ ranks for class m«*ar>g for 
Indl^dufi C uiJcf»rt--type student a3)lnlon of tSBcrdxig Items are conmerec In 
this :i a we r Mthods are g^erally apxllizsoble In any situation in 

whlcr -aarrnDS for ffleacs on Llker^-^ype Items a2:& iff=»sired. The flrsr method 
InvesTXggg^d ^ coianKmly use$ii L±aear Intscsolatlon method In «ilch the 
reports! p^'^rcenti^I^ r^ks (or bsnoa:) are obtaoaec via linear lntecpK3<latlon 
between data points in uae eoipi^<:/^.l cumulative dlffitrlbutlon of clk<ss 
means. Sr. the second oesthod, t^fcc parameters nf a two parameter bet^ dis- 
tribution are Clrst estiaiated "^mnr the distribution of class means and then 
used to sm^^tDa>t'^ rie desired pensntlle ranks. The estimated beta distri- 
bution m»aikc-i stxidji -d becaa^^ If ftrnnd to be J^roprlate, It can be ex- 
pected tc:ija:pv ide a tmoioth» regular function for tiie percentile mks, to 
require the st^ ra^re of -a fraction of the amount of Information re^B^lred 
with the linear interpaiLatlon method » and to provide a simple method for 
accumulating ^4 ^vdatJjzg tlse percentile norms as additional data are gath- 
ered. In adHLttoBc^ thu& estimates may be more accurate than those obtained 
In the usual way. 
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Linear Interpolation Method 

To arrive at norms using either of the two methods. It Is necessary 
to collect data on each of the Items from a sample from the population or 
populations of Interest. When using the linear Interpolation method, the 
sample data (e.g.., class means) are rank ordered and the percentile ranks 
for each of the data points are calculated from this empirical cumulative 
distribution. When the data collected are Item means, which Is the concern 
here, an unlimited number of different means (on a closed Interval bo'\n- 
ded by the scale values) could possibly occur. In order to arrive at the 
percentile ranks for means that were not Included In the sample data, lin- 
ear Interpolation Is generally used. [Note that other methods of Inter- 
polation such as the logistic method described by Marco (1977) could also 
be employed.] The linear Interpolation procedure can be used to arrive 
at percentile ranks of Item means collected In the future. Note^ however, 
that all of the norming data (the entire empirical cumulative distribution) 
must be stored to update the norms using data collected In the future. 

Since the number of classes Included In the Initial norming of stu- 
dent opinion of teaching Items Is usually "small" (typically fewer than 500) , 
the function describing the relationship between the Item class means and 
percentileri that is produced by the linear interpolation method is very 
irregular. When graphed, it often appears as a very Jagged function — at 
least in some scale regions. In course ratings, the irregularities will 
tend to be more pronounced at intervals where the data are sparse and are 
especially notable at the lower tail of these (usually) negatively skewed 
distributions. It seems reasonable to assume that the "actual" (population) 
distribution function would be a smooth, regular function. 
Estimated Beta Distribution Method 

One way to arrive at a smooth and regular function is to estimate the 
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parameters of an appropriate family of theoretical, continuous ^Ustv^Lbotlons 
from which percentile ranks could be derived. If a particular — ly off 
distributions Is appropriate, the resulting percentile ranks «^Uld .em to 
be more accurate than those derived using the linear Int erp a itfff loyn m njod. 

Since the responses to the student opinion Iteans Investi^^J^d v^iere 
were bounded [ranging f rom cme (strongly disagree) to six (st5*:^^*qgly a^ee)], 
a bounded family of distributions seemed most useful tio adequate* / rnooei 
the data. The use of an lumounded distribution, such as the i :3al, wrrrid 
assume that class means might range from nslnus to plus lnf±nl ac imnos- 
slblllty with Llkert-type data, and necessarily Introduce soc^ - rrrrrrDn 
at both ends of the scale. 

The beta family of distributions Is bounded and can tafc on wldte 
variety of shapes. For this study, the lower bound of the. bet%" was Fixed 
at a value of one (strongly disagree) and the upper bound at a v^Iloc tt 
six (strongly agree) . The two necessary parameters estimated) 
data allow the density function for each Item to be symmetry hly 
skewed In either direction, or J-shaped, as well as other i '-^te 
shapes. Since the distribution of means on the student oplnl Aii^ 
Investigated here are typically bounded with skewed shape, tb yr* \ faaiily 
would seem to be an Ideal family of distributions to use a^ del lor 
the data. 

The method chosen for estimating the two parameters cz oeta dis- 
tribution was the method of moments (Hogg and Craig, 1970) wile a max- 
imum likelihood procedure would probably provide for more agr.; '?ate esti- 
mation of the beta parameters, the fact that it requires sbl tessrative solu- 
tion (Johnson & Kotz^ 1970) and that it requires tthe storage xf ^ a large 



^Actually, to achieve appropriate scale bouniis,. the probafcLllity 
density funtion was f (ax + b). 
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amount ef SRbzmatlon in order to accumulate norms over timSy caused us to 
rale amt likelihood as a viable estimation procedure In the present 

sit uaiL-iuRu In circumstances ^rmltting tts use» however, some addlrlOEial 
gair Ic ^emacy might be expected, 

lU^ Twrfao H of moments recpilres only tdrr&si gtatlstlTrs (the nirmtier of 
liMi ■ ■Jt.infl , the mean of the class means, ^nc: me vaTi»>nce of the class 
anssa' estimate the two becs parameters The veuin md variance of the 
C: jam lOucSa MTe equated to the first and siecand HMUe^ at&r of the beta dls^ 
trrlb dLvf^n-ria algebraic formulas in order to esiltiui -3 r±e two beta para- 
I itez : (n:x and B) • 
UpdaiiliiA-^ and Accumulating Norms 

t i .s often desirable to accumulate and la^-iane noiEms over time. That 
' . , as ~!4&07 data are collected they can be used- in the estimation of percen- 
tile ranks. Assuming that the characteristiGEr of the population of inter- 
est do Aat change substantially over time, the inclusion of additional data 
into the procedure for calculating percentile ranks would be expected to 
fncreaise the precision of the estimated percentile ranks regardless of which 
method i& used. 

The accumulation of data and the updatcaag of percentile ranks over time 
using ::he linear interpolation method requrras that all of the previously 
collected data be stored and used in combization with the newly collected 
data. In order to accumulate data over time, using the estimated beta 
distribution method, it is necessary to starr: only the number, mean, and 
variance of the class means. As additional ^0s&a are collected, the beta 
paraiaeters and percentile norms can be re-estdnttted by considering only 
the number, mean, and variance of the previouaiy collected data and by 
updating these values using the new data. This, if the estimated beta 
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distribution netsmd is found to Isl. appropriate, it woii:±d require the star- 
age of only a xslatively small amacmt of infonaation -50: each item and pro- 
vide a simpli^ ^sethod for accinrmLat^ng percentile norss as^ desired. 

METHODS rjm ESTIMATING ^filD ACCUMULATING PERCEBTnzZ RANKS 

l ~,zrteaT InteriBcIianioa Method 

Estimatimi . '^ne percentile rank of an item mean 3S(X)] contains:: in 
tse initial u«...i iii1^=iir data set was calculated as, 

PR(X) = Pet Below X + |(Pct at X). 7-1) 
Linear interpola tion was used to estimate the percenril p ranks for Item 
means between thw^Be item means contained in the initial norming data set. 

Accumulating data . When new data are encountered, the percentile 
ranks from the i itial norming data could be used to estimate the percen- 
tile ranks for c new data. If a sizable number of new means were col- 
lected, it mig^• be desirable to include the new data to produce an updated 
set of estimated percentile ranks. That is, as new data are accuinulared, 
they could be c-^mbined with the previously collected data to produce up- 
dated percentile ranks. Note that this requires the storage of all new 
and previously collected data. 
Estimated Beta Distribution Method 

Estimation . The formula of the beta density function of y, where y 
ranges from zero to one is, 

f (y) - ^(3^ y''"^(l-y)^"^(0<y<l). (2) 

In this formula a and 3 are the parameters which define the shape of the 
beta density function and were estimated via the method of moments in the 
present study. The B(a,3) expression is the beta function (Novick Jack- 
son), 1974) and is a scaling constant* 

ERIC 



The a and 3 parameters as estimated the method of moraears for 
a variable ranging from zero to one are 

a « M^(l-M)/V-M (3) 

amd 

e =» M(l-M)/V-a-£^ (4) 
Tm these equations, M Is the sample mean am: V the sample variance of a 
variable ranging from zero to one. 

Although a more general form of the heta density function which In- 
cludes upper and lower bound parameters (m^ constants) exists, the form 
of the beta density function In (2) was u«sec for simplicity. To use (2) 
In the present study. It was necessary t- subtract one from each class 
mean and divide the resulting value by five In order to relocate the class 
means on a [0,1] Interval. The mean ann variance of the relocated values 
were used to estimate ot and 3 by (3) anc (4) . Percentile ranks of the re- 
located means were estimated by Integrarlng (2) over the appropriate limits 
and multiplying the resulting value by 100. The integration was completed 
with the locally developed program listed in rhe Appe-^iv' : although the IMSL 
(1978) packaged routine, MDBETA, could have also been used. (The locally 
developed program was used because it was found to be more efficient.) The 
zero to one scale was then transformed back to a one to six scale while 
keeping the associated percentiles intact. 

Accumulating data . The only summary statistics from the initial 
norming which are stored in order to accumuiatf. data and produce updated per 
centile ranks are the number of class means, the mean of the class means, 
and the variance of the class means from the initial norming sample. The 
number of class means, the mean of the class means, and the variance of 
the class means for a new sample can be pooled with the statistics from the 
initial norming to produce an updated number of class means, mean of class 
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means, variance of class means. Updated a and 3 parameter estimates 
can ^h0wr 96 calculated using (3> and (4) with the updated statistics and 
the unKSffzsHi percentile ranks can be calculated using the updated estimates 
of a 

The updating procudures for the two methods were not evaluated in the 
prestent paper except for their relative ease of implementation. The proce- 
dures have been included for the purpose of indicating how they might be 
accoonplished . 

EVALUATION METHOD 

DaLrB Source 

The data for the initial beta fitting and comparisons (the initial 
saanpling) were gathered by administering each of 80 student opinion of tea- 
ching items in a representative sample of 189 classes at The University of 
Iowa in Spring, 1974. The 80 items were sain)led from the general areas of 
course content, objectives, and structure; instructor's behavior; instruc- 
tional methods and materials; and outcomes of instruction. Each of the 
items was scored on a Likert scale ranging from one (strongly disagree) 
to six (strongly agree). 

Fourteen of the initial norming items were administered in a different, 
but comparable, sample of 210 classes at The University of Iowa in Fall, 
1975. These data were used as a cross-validation check of both of the per- 
centile estimation methods. 

Details regarding the methods used to secure participation of instruc- 
tors, documentation of the representativeness of the sample, and items used 
are provided in two local publications for interested readers [Evaluation 
and Examination Service Summary Report No. 44 (1976) and Evaluation and 

er|c . 10 
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Examination Service Memo No. 26 (1978)]. The numbers used to Identify the 
Items In the present paper correspond to the Item Identification numbers 
In the latter report. 
Procedure 

Percentile ranks for each of the 80 items included in the initial 
normlng study were estimated using each of the two methods. The fit of the 
beta to the initial sample data was evaluated using a Kolmogorov (Conover, 
1971)-like statistic. Let beta (X) be equal to the value of the beta 
(cumulative) distribution function at X and P(X) the percentile rank at X 
defined in (1) and divided by 100. The statistic T was defined as, 

T= supdif I beta(X)-P(X) |. 
X 

That is, for those means contained in the initial normlng sample, T is 
the greatest absolute difference between the percentile rank of a mean 
computed from (1) and the percentile rank of a mean estimated using the 
estimated beta distribution method divided by 100. Note that T is not a 
true Kolmogorov statistic and that the critical values used in this paper 
are probably too large to fully control the significance levels at the 
Intended values ^ . 

Two additional Kolmogorov-like statistics were calculated for each of 
the 14 items included in the cross-validation sample. The statistics were 
designed to evaluate which of the t^-o methods produced percentile ranks 
that compared tiost closely to percentile ranks which were calculated by 
applying (1) to the cross-validation normlng data only. That is, the 



^The statistic T is not a Komolgorov statistic because percentile 
ranks instead of observed cumulative distribution values were used, 
because there were a large number of "tied" means, and because the beta 
parameters were estimated from the data. Noether (1967) has indicated 
that a large number of ties leads to a conservative test. The use of 
the data to estimate the beta parameters would also be expected to lead 
Q to a conservative test. 
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statistics calculated from the cross-validation data were used as a 

criterion to evaluate the relative accuracy of the two methods. 

For the cross-validation data, let be a mean observed in the 

cross-validation sample and P (X ) be the percentile rank of X divided 

c c c 

by 100 that was calculated from the observed cross-validation data using 

(1), Let beta^CX^) be the value of the beta distribution function which 

was estimated from the initial norming data and evaluated at X . Let 

c 

Pj(X^) be the percentile rank of estimated using the linear interpola- 
tion method with the initial norming data. The test statistics are then, 

Ti = supdif I P (X )-P (X ) I, 

X J- c c c 

and ^ 

Ti = supdif I beta^(X^) - P^(X^) |. 

I c c c ' 

c 

For the means observed in the cross-validation sample, Ti and T2 are the 
greatest absolute difference between the percentile rank of all of those 
means estimated from the cross-validation sample only and the percentile 
rank of that same mean estimated using the linear interpolation and beta 
distribution on the initial sample data after the percentile ranks are 
divided by 100. 

Smaller values of Ti and T2 indicate a closer correspondence between 
the percentile ranks estimated from the initial sample by the respective 
method and the percentile ranks observed in the cross-validation. The 
linear interpolation method will be judged to provide for a more accurate 
representation of the cross-validation data for those items where is 
smaller than T2* The estimated beta distribution method will be judged to 
provide for a more accurate representation of the cross-validation data for 
those items where T2 is smaller than T^. The values of the suranary statis- 
tics Tj and were compared to one another and not to Kooolgorov 

12 
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critical values. 

Goodness of fit comparisons considered thus far apply only to those 
situations in which the initial normlng sample size is near 189. To allow 
greater generalizability of results, random samples of the class means 
from the initial sample were drawn for each of the 14 items that were also 
included in the cross-validation norming. For each of the 14 items, ten 
random samples were drawn with each of the probabilities of selection of 
•10, .25, .50, .75, and. 90 for each class mean observed in the initial nor- 
ming. The linear interpolation and estimated beta distribution methods 
were used to estimate percentile ranks and the Tx and T2 statistics were 
calculated using the cross-validation data. 

RESULTS 

Initial Normlng Data 

The number of means, the mean and variance of means, the Ol and 3 
values for the estimated beta distribution method, and the value of the 
test statistic T are presented in Table 1. The item numbers in Table 1 
correspond to the item ideiitif icatioti tiusibers lis ted in the Evaluaticn 
and Examination Service Memo No. 26 (1978). 

Those items for which the T statistic surpassed the a".05 critical 
value for a two-tailed Kolmogorov statistic are noted. Had the Kolmogorov 
test been strictly appropriate, four of the 80 T statistics (80 X .05 « 4) 
would be expected to surpass the critical value if each of the beta dis- 
tributions fit the corresponding population data. Since the Kolmogorov 
test is probably conservative in the present situation, fewer than four 
of the T statistics would be expected to surpass the a«.05 Kolmogorov 
critical value If each of the beta distributions fit the corresponding 
data. As noted in Table 1, eight of the T statistics surpassed the a". 05 

ERIC 
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Kolmogorov critical value. This Indicates that the beta family did not 
fit the population distribution for some Items. However » the fit appears 
to be reasonable with the laedlan value of T being .0715. That ls> the 
greatest absolute difference between the observed percentile ranks and 
estlmatp^ beta distribution method percentile ranks, after the percentile 
ranks were divided by 100» was no more than .0715 for half the It^s and 
no more than .1346 for any Item in the set. ' 

Insert Table 1 About Here 

Cross-Valldatlon Data-Complete Sample 

The number of means » the mean and variance of the means, and the Ti 
and T2 statistics for the cross-validation normlng are presented In Table 
2. Unfortunately 9 the Interpretation of these results Is ambiguous. The 
Ti statistic was smaller than the T2 statistic for 10 of the 14 Items 
(p<.18 for a two-ended sign test) suggesting that the linear Interpolation 
method may be more accurate. However, the median Ti statistic (.0977) was 
larger than the median T2 statistic (.0944) suggesting that the estimated 
beta distribution method was more accurate, 'flius, we conclude that when 
the Initial normlng sample size was In the neighborhood of 189, both meth- 
ods appeared to be nearly equally accurate for estimating percentile ranks 
for the student opinion of teaching Items studied here. 

Insert Table 2 About Here 

Graphical Comparisons . Graphs of the fitted beta, the Initial normlng 
percentile ranks and Interpolated percentile ranks, and the cross-validation 
data for the 14 Items Included In the cross-validation normlng are provided 
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in Figures 1-14^. Overall, these figures illustrate the smoothness and 
regularity of the linear interpolated graphs. Visual inspection reveals 
that the fit of the beta to the initial nonaing data was probably as 
good as could be expected from a class of bounded curves for which only 
two parameters are estimated from the data. Thus, the beta appears to be 
an acceptable modeling distribution for the student opinion of teaching 
items which were studied. 



Insert Figures 1-14 About Here 



Evaluation of the differences between the two methods may be facili- 
tated by referring to the graphs for which the Ti and T2 statistics dif- 
fered by the greatest margin. The item (number 108) for which the differ- 
ence between Ti and T2 was the largest and favored the linear interpola- 
tion method is shown in Figure 1. The percentile ranks produced by the 
two methods for item 108 differed by the greatest amount in the neighbor- 
hood of a scale value of 5. The cross-validation normlng data was closer 
to the linear interpolated percentile ranks than to the estimated beta 
distribution percentile ranks in this interval. It appears to be the 
case that with this item, the estimated beta distribution method smoothed 
out an apparent irregularity which was actually a real bend in the distri- 
bution function. This illustrates, of course, the danger in assuming the 
degree of irregularity Implied in the beta model. The beta cannot provide 

The linear interpolation method results are represented by a series 
of 'o* symbols and interpolation between points indicated by a solid line. 
The estimated beta distribution method results are represented by a smooth 
solid line and the cross-validation results are represented by a series of 
'+* symbols. 



EKLC 



15 



- 13 - 



for an adequate fit to this item because the actual distribution function 
apparently has more than one point of inflection. 

The data for the item (number 241) for which the difference between 
Ti and T2 was the largest and favored the estimated beta distribution 
method are graphed in Figure 6. For this item, the percentile ranks pro- 
duced by the two methods differed most in the neighborhood of a scale 
value of 5. The "jump" in the graph of the linear interpolation method 
for the initial norming data occuring near 5 was smoothed out by the esti- 
mated beta distribution method. For this item, a large irregularity pres- 
ent with the linear interpolation method was presumably due to sampling 
error because the irregularity was not present in the cross-validation 
data and was the reason that the estimated beta distribution method was 
found to be substantially more accurate. 

The data for items 108 and 241, and to a lesser extent the data for 
the other items, illustrated in Figures 1-14 tend to support the conjec- 
ture that the estimated beta distribution method smooths out irregularities 
and in cases where those irregularities are due to sampling error » the 
estimated beta method appears superior. In other cases, the beta seems 
to smooth out apparent irregularities which were also present in the cross- 
validation data, and thus, may not be due to sampling error. If this 
conjecture holds, a procedure for identifying those items for which the 
estimated beta distribution may not provide adequate estimates would be 
desirable. Items identified by this procedure could be closely monitored 
and, if necessary, the linear interpolation method used for estimating 
and accumulating percentile norms for these items. A method based on 
cross-validation data could be used for the assessment although no speci- 
fic method is proposed. 
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The analysis of the data with the complete set of 189 means from 
the initial :Bainple suggests that each method was better in some instances 
but that, overall, the methods produced nearly equally accurate results. 
These data do not indicate which method will be expected to be most accu- 
rate when other sample sizes or other kinds of items are used. The next 
set of results address this question of the comparative accuracy of the 
methods for smaller sample sizes. 
Cross Validation Data — Small Samples 

The median Ti and T2 summary statistics for each of the lA items at 
each sample size as well as the median Ti and T2 summary statistics over 
all of the items are presented in Table 3. As expected, there was a ten- 
dency for the accuracy of both of the methods to decrease as the sam- 
ple size decreased. This tendency was especially pronounced at the small- 
est sample sizes used in the study (N - 19 and 49 classes) • 

Consistent with the results from the full sample sizes presented in 
Table 2, the median value of Ti over all items was greater than the median 
value of T2 over all items at each sample size. This result provides one 
indication that the esciaaced beta distribution method produced more accu- 
rate percentile norms than did the linear interpolation method. 

Insert Table 3 About Here 



The proportion of the 10 samples at each item and sample size combi- 
nation as well as the proportion over all items at each sample size for 
which the individual (not median) value of Ti was greater than T2 is also 
shown in Table 3. Note that these values indicate the proportion of sam- 
ples for which the estimated beta distribution method provided a closer 
fit to the cross-validation data than did the linear interpolation method. 
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For smaller Bamp^Le sizes (say 50 or fewer class means), the estimated 
beta distribution results corresponded more closely to the cross-valida- 
tion data than £Ld the results from the linear interpolation method, 
but this does not necessarily hold for larger sample sizes. This con- 
clusion holds whether the median values of Ti and T2 or the proportion 
of times that the Ti statistic was greater than the T2 statistic was used 
to evaluate fit. 

In order to be able to make more precise statements, an analysis of 
variance was completed on the individual Ti and T2 values. The main effect 
were item (14 levels) which was considered as a between random effect, 
sample size (5 levels) which was considered as a between fixed effect, 
method (2 levels) which was considered as a within fixed effect, and sam- 
ple (10 levels) which was considered as the random "subjects" effect. The 
ANOVA summary table, including estimated variance components, is shown in 
Table 4. The notation used to indicate main effects and interactions is 
that of Myers (1972) . All effects and interactions tested surpassed the 
.01 critical value except for the methods main effect. Since item and 
sample were considered to be random effects- their main effects and inter- 
actions will not be discussed further. 

The size, method, size x method, and overall means are shown in Table 
5. Of special Interest is the size x method interaction. Paired compari- 
sons of nBthod means at each level of sample size indicate that the methods 
differed anly at the two smallest sample sizes. The difference was in 
favor of the estimated beta distribution method. Thus, with sample sizes 
of around 50 or fewer classes, the estimated b'eta distribution method was 
found to be superior. For the larger sample sizes studies, the two methods 
produced results which were about equally accurate. 
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DISCUSSION 



The estimated beta distribution and linear interpolation methods for 
estimating percentile norms were found to provide nearly equally close 
approximations to the observed percentile ranks of the cross-validation 
data for the large initial sample sizes. For initial samples of 50 or 
fewer class means, the estimated beta distribution method was found to 
produce more accurate results than the linear interpolation method. 

For smaller sample sizes, the greatest accuracy was achieved by the 
estimated beta distribution method. Even for the larger sample sizes 
which were studied, the estimated beta distribution produced reasonably 
accurate results which were judged to be as accurate as the results pro- 
duced by the linear interpolation method. Thus, the estimated beta dis- 
tribution method would be expected to produce reasonably accurate results 
for the kinds of items from which the student opinion of teaching items 
included in this study were selected. 

In addition to providing reasonably accurate results, the estimated 
beta distribution method may prove less costly and easier to use than the 
linear interpolation method in many situations. The estimated beta distri- 
bution method requires the storage of only a small fraction of the amount 
of information required with the linear interpolation method when the per- 
centile norms are to be used in the future. For example, the University 
of Iowa cafeteria system for collecting student opinion of teaching data 
contains 200 items. If there were, on the average, 100 class means for 

^It would be expected that, unless the beta model fit the population 
data perfectly (which is unlikely), at some sample size greater than 189 
the linear interpolation method would produce more accurate results because 
it allows for a much wider (nearly infinite) range of forms for the func- 
tion relating percentiles to percentile ranks. 
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for each Item^ then 20,000 items X mean // class means) of data would 
need to be stored and analyzed each semester if the linear interpolation 
method were used. As more class means were used to update the percentile 
norms, even more than 20,000 pieces of data would need to be stored and 
analyzed, even if the number of items remained constant. In contrast, 
only 600 (# items X 3) pieces of data would need to be stored with the 
estimated beta distribution method. This figure would not increase tmless 
more items were added to the item pool. Note that rounding the item means 
to, say, the nearest tenth or using stanines could decrease the storage 
requirements for the linear interpolation method. However, this would 
still not reduce the storage requirements to three summary statistics per 
item, the number required with the estimated beta distribution method. 



CONCLUSION 

The beta family of distributions was found to provide reasonably 
accurate estimations of percentile ranks for class means arising from the 
use of Likert-type student opinion of teaching items. The estimated beta 
distribution method, in addition to providing as accurate or more accurate 
norms, produces a smoother function, requires the storage of less infor- 
mation, and provides for a method for updating and accumulating norms which 
requires substantially less storage of data than the linear Interpolatd^on 
method. Based on these considerations, we are presently using the sttdanted 
beta distribution method for creating, updating, and accumulating nmrms for 
the student opinion of teaching items which are administered at the Un±yer- 
slty of Iowa. The estimated beta distribution method is especially recom- 
mended for accuracy purposes with sample sizes less than 50 class mesms as 
well as for simplicity and cost reasons when percentile norms are to be 
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updated and accumulated. 

The estimated beta distribution offers the potential for improving both 
the accuracy and the economy of the student rating 5Uii-ary results reported 
to faculty members. Since these systems are ofte- -y uaed and results 

are frequently considered in critical promotion. nd tenure decisions. 

Improvements in either respect over the methods r itly used could be of 
substantial importance in the retention of faculty seen (by students) to 
be good or poor teachers and in the modification of teaching methods based 
on student opinions. More generally, the close fit of the beta to the 
data indicates that the beta should be investigated as a possible model 
for aggregate statistics (e.g., means, medians) arising from the use of 
Likert-type items in other applications. 
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TABLE 1 



Item Statistics For Class Means Initial Noming Study (N » 189 classes) 



Item 


Sample 


Statistics 


Estimated Beta 


Parameters 


J 
T 


Mean 


Variance 


Alpha 


Beta 


101 


4.64 


0.70 


4.41 


1.65 


0.1034* 


102 


4.60 


0.77 


4.01 


1.56 


0.0797 


103 


4.88 


0.61 


4.77 


1.37 


0.0588 


104 


5.06 


0.55 


4.81 


1.12 


0.1057* 


108 


4.66 


0.74 


4.09 


1.50 


0.0628 


111 


5.06 


0.33 


8.45 


1.96 


0.0504 


112 


4.54 


0.61 


5.24 


2.16 


0.1014* 


114 


4.63 


0.57 


5.56 


2,10 


0.0738 


115 


4.72 


0.60 


5.19 


1.79 


0.il29* 


116 


4.91 


0.46 


6.42 


1.80 


0.0785 


117 


4.83 


0.76 


3.73 


1,13 


0.0900 


122 


4.62 


0.58 


5.51 


2.11 


0.0699 


123 


4.72 


0.52 


6.03 


2.08 


0.1043* 




4.85 


0.70 


4.31 


1.28 


0.0462 


127 


5.16 


0.56 


4.35 


0.88 


0.0738 


128 


4.72 


0.74 


4.07 


1.40 


0.0713 


129 


4.91 


0.47 


6.33 


1.77 


0.0967 


130 


5.27 


0.33 


7.35 


1.26 


0.1027* 


201 


4.83 


0.62 


4.79 


1.46 


0.0782 


203 


4.26 


0.84 


3.74 


1.99 


0.0469 


206 


4.62 


0.65 


4.88 


1.86 


0.0572 


207 


5.00 


0.37 


7.81 


1.95 


0.0535 


208 


5.31 


0.44 


5.01 


0.81 


0.0726 


209 


5.43 


0.36 


5.29 


0.67 


0.0696 


210 


4.82 


0.66 


4.41 


1.36 


0.0595 


211 


4.56 


0.78 


3.94 


1.60 


0.0640 


212 


4.81 


0.54 


5.64 


1.76 


0.0668 


214 


5.17 


0.42 


6.12 


1.22 


0.0715 


215 


4.74 


0.60 


5.15 


1.74 


0.0780 


216 


4.85 


0.50 


5.99 


1.80 


0.0881 


217 


5.21 


0.5S 


3.91 


0.73 


0.0866 


218 


4.64 


0.73 


4.18 


1.56 


0.0543 


219 


4.64 


0.72 


4.27 


1.59 


0.0595 


221 


5.10 


0.52 


5.05 


1.12 


0.0682 


226 


5.04 


0.51 


5.27 


1.25 


0.0605 


228 


5.10 


0.54 


4.78 


1.05 


0.0805 


233 


4.58 


0.61 


5.25 


2.08 


0.0782 


234 


4.52 


0.79 


3.92 


1.65 


0.0717 


239 


5.02 


0.49 


5. 65 


1. 37 


A AT CO 

0.0759 


241 


4.84 


0.47 


6.57 


1.99 


0.1346* 


242 


4.84 


0.73 


3.92 


1.19 


0.0455 


245 


4.70 


0.87 


3.37 


1.18 


0.0417 


247 


4.90 


0.68 


4.13 


1.17 


0.0809 


301 


4.90 


0.58 


4.94 


1.38 


0.0775 


303 


4.24 


0.82 


3.85 


2.09 


0.0897 


304 


4.68 


1.07 


2.59 


0.92 


0.0783 


305 


4.33 


1.00 


3.05 


1.53 


0.0540 
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TABIE 1 (cont.) 



Item 


o ample 


oCaciscics 


iLSi.imai.Ba ue ta 


^ o T* o mo ^ o 1* Q 




Mean 


Variance 


Alpha 


Beta 


'inn 




U. )7J 




1 91 








U. /z 


'I ft? 

J . oz 


1 07 


n nA9n 


312 


H» Jl 


U. OD 


J. OO 


1 ftft 
L . OO 


U . U J JX 


QIC 


H m Ol 


U. oo 




1 'X^ 


U . U J07 


317 


D m 1/ 


0. 31 


O. 31 


1 AO 

1 . oy 


ft ftOfifi 

u. uyoo 


318 


J. 12 


0.33 


Q on 


1 "7 A 

1. 7o 


ft ftilQ/: 

u. uoyo 


321 


4.86 


0.66 


4. 35 


1. Zo 


ft ft O A 1 

u. oy4i 


322 


4.78 


0.45 


C ft ^ 

o. 9o 


O OA 

2 .24 


C\ ftOl Q 

u. oyio 


326 


4. 64 


0. 75 


A ft A 

4. 04 


1 C ft 

1.50 


ft ft A A C 

U.04o5 


329 


4.37 


1. 02 


O ft^ 

2. 96 


1 AO 

1.43 


ft ftA 1 1 

U. Ooll 


332 


4.77 


0. 73 


A m 
4. 0/ 


1 o o 
1. 33 


ft 1 1 ftft^r 

u. iiyu'^ 


333 




0. 50 


o. 53 


O AA 
Z . OO 


ft ftAAA 
U • U004 


402 


4. 94 


0.68 


/ 1 ft 
4. 10 


1 1ft 

1. 10 


ft ryn Q t 

U. U7ol 


403 


4.28 


0.97 


3. lo 


1 AT 

1. o7 


n ft A 7 7 

U. U4/ / 


404 


4. U4 


O ft Q 

0. 99 


Q nc 


1 07 

1. y / 


ft HA OA 


405 


4. 44 


n "7 c 
0. /5 


A 9 7 


1 07 


n nft^ft 

U • UO JO 


407 


5.26 


0. 30 


Q in 
o. lU 


1 A 9 
1. 4Z 


ft nft*? 7 

U • UOJ / 


408 


4. yy 


n CO 

u. jy 


A Aft 


1 10 

X . xy 




409 


4. 4o 


U. 55 


J . yy 


9 AA 

z . oo 


n n77A 


410 


4. 


U. D/ 


A ft*; 

Oj 


9 AA 
Z . 40 


n n7nA 


411 


4 . yo 


0.4/ 


A 1 Q 

o . ly 


1 A7 
X . 0 J 


n nA70 
u • uo / y 


413 


4. / J 


n Ail 
U. 4d 


7 n9 
/ . uz 


9 An 
Z . 4U 


n n77ft 

U • U/ JO 


414 


4. 3Z 


0. 76 


A 9n 
4. zu 


9 19 
Z . Ij 


n 117 ft 9 
u • u/ oz 


415 


4. 75 


n A 
0. 4d 


A o<; 
0. yo 


9 71 
Z . JX 


n nsAA 

u • U J^O 


416 


4.87 


0.43 


7.10 


2.06 


0.0588 


417 


4.38 


0.57 


5. 78 


i:. /o 


A /\ /. O /\ 

U. 


418 


4.46 


0.80 


3.90 


1.73 


0.0529 


/. o/\ 


4. 60 


0. 66 


4.74 


1.B4 


0.0582 


423 


4.51 


0.66 


4.82 


2.04 


0.0798 


424 


4.72 


0.66 


4.61 


1.58 


0.0568 


425 


4.69 


0.57 


5.54 


1.96 


0.0798 


427 


4.60 


0.68 


4.60 


1.79 


0.0660 


428 


4.23 


0.94 


3.27 


1.79 


0.0432 



*Indicates lack of fit of the estimated beta distribution method esti- 
mate to the results from the linear interpolation method at approximated a" .OS. 

#T is the greatest absolute difference between the percentile rank of a 
mean from the initial sample data using the linear interpolation method vs. 
the estimated beta distribution method. 
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TABLE 2 

Summary Statistics and Test Statistics 
for Cross-Validation 
(N B 210 classes) 



Cross-validation Test 



Item Statistics Statistics 





Mean 


Variance 


••• 1 


To 
•••2 


108 


4.62 


0.57 


.0798 


.1104 


203 


4.19 


0.70 


.0851 


.1017 


207 


4.67 


0.57 


.1995 


.2194 


209 


5.46 


0.28 


.0945 


.0629 


221 


4.91 


0.51 


.1552 


.1767 


241 


4.86 


0.51 


.1139 


.0389 


245 


4.78 


0.62 


.0778 


.0852 


306 


4.63 


0.79 


. .0638 


.0678 


402 


4.83 


0.54 


.1246 


.1430 


405 


4.40 


0.62 


.1080 


.0667 


407 


5.06 


0.26 


.2139 


.2341 


415 


4.80 


0.35 


.0969 


.0671 


416 


4.87 


0.39 


.0791 


.0883 


417 


4.26 


0.57 


.0985 


.1004 


Median 






.0977 


.0944 










.29 



^Proportion of the 14 items for which Ti > Ta , that is 
for which the estimated beta distribution method was 
superior* 



or 



TABLES 

Median Values of Ti and T2 at each of Five Sanple Sizes and Proportion of Times Ti > T2 



Senile 

,%mm .75(M^142) .5Q(N^94) .25(N^47) >10(N''19) 

Statistic 



12 Ti T2 Ti T2 Ti T2 Ti T2 P^ 



108 .0820 .1128 .00 .0831 .1017 .00 .0998 .1144 .10 .1373 .1422 .30 .2236 .1187 .50 

203 .0915 .1122 .00 .0870 .1063 .00 .1048 .1110 .30 .1394 .1150 .30 .2154 .1158 .60 

207 .2068 .2217 .00 .2023 .2129 .00 .1937 .2184 .20 .2180 .2254 .40 .2446 .2118 .90 

20« .0894 .0610 1.00 .1008 .0640 1.00 .0928 .0615 1.00 .1023 .0764 .70 .1573 .1018 .70 

221 .1626 .1747 .00 .1606 .1749 .00 .1618 .1802 .10 .1583 .1676 .30 .2124 .2187 .60 

241 ' .1155 .0382 LOO .1027 .0428 1.00 .0920 ,0542 .90 .1276 .0646 1.00 .1233 .1084 .90 

245 .0813 .0870 .20 .0882 .0878 .20 .0848 .0866 .50 .1093 .1189 .20 .1754 .1391 .90 

306 ,0657 .0720 .10 .0718 .0765 .30 .0843 .0909 .30 .1236 .0715 .90 .1371 .1277 .50 

402 .1235 .1380 .00 .1050 .1262 .00 .1211 .1450- .20 .1139 .1335 .30 .1908 .1825 .50 

405 .1046 .0742 1.00 .1084 .0759 .90 .0881 .0641 1.00 .1130 .0991 .50 .1852 .1321 .70 

407 .2151 ,2361 .00 .2284 .2463 .10 .1974 .2246 ,20 .2820 .2486 .40 .2168 .2200 .40 

415 .1052 .0676 1.00 .1151 .0757 .90 .1029 .0808 .90 .1297 .1030 .80 .1657 .1580 .90 

416 .0812 .0885 .10 .0793 .0823 .50 .0818 .0962 ,40 .1116 .1175 .40 .1427 .1189 .80 

417 .1034 .0989 .40 .0842 .0805 .60 .1141 .0998 .50 .1617 .1320 .70 .1441 .1153 .90 

Urn .1040 .0937 .34 .1018 .0851 .39 .1014 .0980 .47 .1286 .1182 .51 .1803 .1299 .70 



^Proportion of the random samples for which Ti > T2 , that is for which the estimated beta distribution 
method was superior. 
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TABLE 4 

AHOVA Summary Table for Index Values 



Source 



Between 



Item 

Sample Size (Size) 
Item X Size 
Sample/Item X Size 



Vlthin 



df 



Method 

Item X Method 
Size X Method 
Item X Size X Method 
Sample X Method/Item X Size 



Total 



699 

13 
4 

52 
630 

700 
1 

13 
4 

52 
630 

1399 



Mean 
Square 



0,2112 
0.1884 
0.0051 
0.0028 



0.0354 
0.0099 
0.1258 
0.0013 
0.0006 



Mean Square 
Ratio 



76.32* 
36.73* 
1.85* 



3.58 
15.67* 
10.06* 

1.98* 



Estimated ^ 
Variance Components 



19.87 
5.24 
0.88 



0.18 
0.86 
0.32 
0.23 



* p<.01 

^Actual values ■ entries X 10 
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TABLES 

Table of Means for the Method X Sample Size Interaction* 



Method 






Sample 






All Samples 


.90(N-170) 


.75(N=142) 


.50(N=94) 


.25(N=47) 


.10(N=19) 


Linear Interpolation 
Beta 


.1152 
.1133 


.1145 
.1125 


.1202 
.1176 


,1438 
.1331 


.1912 
.1581 


.1370 
.1269 


All Methods 


.1142 


.1135 


.1189 


.1384 ■ 


.1747 





*Tukey critical difference for comparisons between any two means contained in the method X size 
interaction is .0112. 
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Appendix 
FORTRAN FUNCTION BCDF 



Function 3CDF(X) evaluates the Integral of the beta distribution with 
parameters A & B to produce 



BCDF(X) - /B(A,B)y^"''"(i-y)^"''"dy. 



The A and B parameters are passed to BCDF In the common statement 



COMMON/BCDFC/RESET.A.B. 



Since there Is a fair amount of computation Involved when A & B change values, 
this program will perform those computations only If RESET -.TRUE.. (Thus, 
RESET should equal .TRUE, for the first call.) After each evaluation of the 
beta, RESET will be automatically set to FALSE.. When A and/or B change 
values, the user should set RESET = .TRUE.. Note that this routine requires 
that, the variable, y, be re-scaled to a zero-one scale. A listing of the 
program, follows. 



POHCTIOM BCOF(Z) 

lOICTIOl BCDF HAS HBITTBN BT ilLLI&H H. SMITH 
APPBOZIBATIOMS PBOH ABBAHOHICZ AND STEGOM ( 1965) HARDBOOK OF 
HATHEHAIICAL FUMCTIOHS. 

COHPUTBS THE IMCOBPLETE BETA IITB6BAL kT 1 BITH PABAHEIEBS A,B. 

TO USE, BEPLACE A ARD B IH THE COBHOH BLOCK HITH APPROPfilATE 
PABAHBTEBS ARD SET BESET TC .TBOE. PBIOB TO THE FIfiST CALL. 

COHBOR/BCDFC/BBSST, AA,BB 

THE F0LL0HIH6 PABAHETBBS ARB COHPOTED OR THE FIBST CALL. ARD 
ABE HOT RECESSABT TO COHPOTE THEBEAFTEB. • . 
LOGICAL BESET 

DATA IRO,A1,A2,A3,C,D1,D2,D3,D4/0,8*0.0/,C0TOFF/0./ 
CHECK FOB FIBST CALL... 
IF (.HOT. BESET) GO TO 10 
A«AA 

B'BB 
IRD«0 

IF EITHEB .?ABAHETBB IS <2, USE BECOBSIOH FOBHOLA. . . 

IF(A.LT.2.)60 TO 30 

IP(B.6T.2< )60 TO 20 CQ 
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c 
c 
c 
c 
c 
c 

C 
C 

c 
c 
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30 XI0«1 

D1=lXP(I1.6iHi(A*B)-iLGiHl(A*1.)-AI.CAM(B)) 

B2»(1*B)/(1*1.) 

03>(A«B4l.)/B 

D4«(A«B4-2.)/(B«1.) 

1«A«2. 

B«B«'2. 
20 COIIIBOE 

1H1«A-1. 

BH1«B-1 • 

ABH2>ABUBH1 

C0T0PF»AHl/iBa2 

6666667* (B-A) /SQHT (ABH2*AHl ♦BHl) 

A2»8.333333B-2*(1./AH1*1./BH1-13./(A*SH1)) 

A3«-5. 333333B- 1*Al* (A2*3./ABH2) 

C«AB1*AL06 (AH1) ♦BHl ♦ALOG (BHl) -ABH2*AL0G (ABH2) 

BESST». FALSE. 
10 If (X.LB.O.)CO TO 60 

IF(X.GB.1.)G0 TO 70 

Y«(2.*(C-AH1*ALOG(X)-BH1*ALOG (1.-X)) ) 

IF(I.LT.O.VI»0. 

Z«SQBT (T) 

IF(X.LT.C0TOFF)2=-Z 

BSF«BCDF-!3989423iEXP^^^^ 
1 )) 

5Ji"Si??;Sm*MA.L)M1.-I)**(B-2.)*^ 

1 -X)))) 
55 IF(BCDF.LT.O.) BCDF«0. 

IF(BC0F.GT. 1.) BCDF'1. 

BETOBH 
60 BCDF*0. 

BET D BR 
70 BCDF=1. 

BETOBH 

BHD 



FUHCTIOI EXP1 (X) 
EXPI'O. 

IF(X.6B.-70)EXP1-BXP(X) 

BBTURM 
EHD 
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ERIC 



